Combining real world imagery with additional imagery from another source requires careful control over which sections of each image are to be used in the final composite image. One common application is to combine images generated by a computer with images acquired from a traditional motion picture, video or digital camera. In order to seamlessly combine the images, the camera parameters from the point of view of the virtual scene camera must be closely matched to the parameters of the live action camera. In addition, for use in on set and live applications, the match between virtual and live action sceneries must be completed quickly, preferably in a matter of milliseconds in order to be useful at standard motion picture and television production frame rates of 24 to 30 frames per second.
Tracking the live action camera and lens may be achieved in a variety of ways. (See U.S. patent application Ser. No. 12/832,480, filed Jul. 8, 2010, entitled “Methods and Systems for Calibrating an Adjustable Lens,” which was published on Feb. 3, 2011 as Publication No. 20110026014, and which issued as U.S. Pat. No. 8,310,663 ('663) and whose entire contents are hereby incorporated by reference.) The measurement of the live action camera's position and orientation may be achieved with optical encoders, by solving the pose of fiducial markers located overhead or in the scene, with inertial accelerometers, by matching high contrast points from one image to the next, and many other methods common in the field.
Real-time measuring methods that measure the angular tilt of the camera (pan, tilt and roll) at the camera itself share a common problem. Small errors in angular measurement become large positional errors when projected out several feet, which is where the majority of live action and virtual scenery seams are located. This can be seen clearly in a composited image as a shifting or sliding between the virtual and live action elements when the camera is moved.
A traditional method of solving this high precision seam is to use post-production software tools, such as Syntheyes, 3D Equalizer, Boujou, PF Track, or Nuke that locate and track multiple high contrast regions from one frame to the next. By analyzing the relative motion of several points on the screen, the live action camera motion and lens parameters can be derived and the motion of the virtual background matched cleanly to the live action foreground.
This process has several limitations. Firstly, if an actor walks in front of the high contrast area being tracked, the camera solution mathematics can easily be disrupted. Secondly, the process of measuring the motion by tracking the individual pixels of the high contrast part of the image is both fragile and time-consuming. It typically cannot be computed in real time, and if a frame of the live action image has a lighting change where the pattern is unrecognizable, the artist must re-specify the high contrast area at the frame of failure to continue the process. In fact, the process of solving an accurate camera track from an arbitrary moving background can exceed the time required to complete the rest of the shot due to the handwork required.
In addition, if the live action camera is zoomed in, the high contrast area that was being tracked can simply disappear from the image, resulting in the camera failing to track the object. It is notoriously difficult for the algorithms used to derive camera intrinsic parameters from shots where the camera is simultaneously changing its focal length (“zooming in”) and translating.
Accordingly, the pure pixel tracking based methods do not work well for the demands of real-time visual effects processing, which must be very rapid to compute as well as robust to the frame-by-frame changes in the live action video image and changes in the adjustment of the cinema lens being used to record the image.
A prior art system is the PREVIZION system, which is available from Lightcraft Technology, LLC. of Venice, Calif. The PREVIZION system includes a camera tracking system, a lens calibration system, a real-time compositing system, and a built-in 3D renderer. An example of a publication disclosing the prior art PREVIZION system is the PREVIZION product brochure, entitled “Previzion Specifications 2011,” published on Apr. 8, 2011, and whose contents are incorporated by reference.